FoleyGAN: Visually Guided Generative Adversarial Network-Based Synchronous Sound Generation in Silent Videos

Deep learning based visual-to-sound generation systems have been developed that identify and create audio features from video signals. However, these techniques often fail to consider the time-synchronicity of visual features. In this paper we introduce a novel method for guiding class-conditioned GAN synthesize representative with temporally extracted information. We accomplish task by adapting synchronicity traits between audio-visual modalities. Our proposed FoleyGAN model is capable conditioning action sequences events leading visually aligned realistic soundtracks. expanded our previously Automatic Foley data set. evaluated FoleyGAN's synthesized sound output through human surveys show noteworthy (on average 81%) performance. approach outperforms other baseline models sets in statistical ablation experiments achieving improved IS, FID NDB scores. analysis showed significance temporal feature extraction as well augmented performance network. Overall, retrieval accuracy 76.08% surpassing existing visual-to-audio synthesis deep neural networks.